The present invention relates to a method of rendering images, in particular to provide occlusion cues, for example in medical augmented reality displays.
Augmented reality (AR) is becoming a valuable tool in surgical procedures. Providing real-time registered preoperative data during a surgical task removes the need to refer to off-line images and aids the registration of these to the real tissue. The visualization of the objects of interest becomes accessible through the “see-through” vision that AR provides.
In recent years, medical robots are increasingly being used in Minimally Invasive Surgery (MIS). With robotic assisted MIS, dexterity is enhanced by microprocessor controlled mechanical wrists, allowing motion scaling for reducing gross hand movements and the performance of micro-scale tasks that are otherwise not possible.
The unique operational setting of the surgical robot provides an ideal platform for enhancing the visual field with pre-operative/intra-operative images or computer generated graphics. The effectiveness and clinical benefit of AR has been well recognized in neuro and orthopedic surgery. Its application to cardiothoracic or gastrointestinal surgery, however, remains limited as the complexity of tissue deformation imposes significant challenges to the AR display.
Seamless synthesis of AR depends on a number of factors relating to the way in which virtual objects appear and visually interact with a real scene. One of the major problems in AR is the correct handling of occlusion. Although the handling of partial occlusion of the virtual and real environment can be achieved by accurate 3D reconstruction of the surgical scene, particularly with the advent of recent techniques for real-time 3D tissue deformation recovery, most surgical AR applications involve the superimposition of anatomical structures behind the exposed tissue surface. This, for example, is important for coronary bypass for which improved anatomical and functional visualization permits more accurate intra-operative navigation and vessel excision. In prostatectomy, 3D visualization of the surrounding anatomy can result in improved neurovascular bundle preservation and enhanced continence and potency rates.
Whilst providing a useful in plane reference in stereo vision environments, traditionally overlaid AR suffers from inaccurate depth perception. Even if the object is rendered at the correct depth, the brain perceives the object as floating above the surface (See for example Johnson L G, et al, Surface transparency makes stereo overlays unpredictable: the implications for augmented reality, Studies in Health Technology and Informatics 2003, 94:131-6; and Swan J E, et al, Egocentric Depth Judgments in Optical, See-Through Augmented Reality, IEEE Transactions on Visualization and Computer Graphics 2007, 13(3):429-42).
For objects to be perceived as embedded in the tissue, our brains expect some degree of occlusion. To address the problem of depth perception in AR, a number of rendering techniques and display strategies have been developed to allow for accurate perception of 3D depth of the virtual structures with respect to the exposed tissue surface. In Sielhorst T, et al, Depth Perception—A Major Issue in Medical AR: Evaluation Study by Twenty Surgeons, Medical Image Computing and Computer-Assisted Intervention—MICCAI 2006 2006:364-72. the issue of depth perception in medical AR has been studied. In agreement with the two references cited above, it was found that depth perception is poor if the AR object is rendered opaquely as it appears to float above the outer body surface even though rendered at the correct depth behind it. Two ways of improving depth perception were identified: rendering both the body surface and the AR object as transparent or rendering the body surface with a window defined inside it such that the window provides an occlusion clue whereby the AR object can be seen within the window but is otherwise occluded by the body surface. Regarding the former approach (transparent rendering), while this may result in improved depth perception for some surfaces, in general rendering two overlayed transparent surfaces results in conflicting visual cues from occlusion such that depth perception can be poor (see for example Johnson et al cited above). The latter approach (rendering a window) has the disadvantage that all information about the body surface within the window is lost.
In Virtual Window for Improved Depth Perception in Medical AR: C. Bichimeir, N. Navab, International Workshop on Augmented Reality environments for Medical Imaging and Computer-aided Surgery (AMI-ARCS 2006), Copenhagen, Denmark, October 2006 (available online at http://ar.in.tum.de/pub/bichlmeier2006windowfbichlmeier 2006window.pdf) various approaches to improving the depth perception obtained with the window approach while maintaining information about the body surface within the window have been studied. The following approaches have been considered: adapting the window shape to the shape of the body surface, rendering the window surface glass-like using highlight effects due to a virtual light source, mapping the window plane with a simple structured texture, simulating a finitely sized frame for the window and setting the background of the AR objects either transparent or opaque. A drawback of all but the last of these approaches is that a 3D model of the body surface must be known so that the window contour or surface can be rendered accordingly. Such a 3D model can be difficult to obtain reliably in particular if the body surface is deforming or changing in other ways during imaging.